Decentralized Federated Learning With Unreliable Communications

Decentralized federated learning, inherited from decentralized enables the edge devices to collaborate on model training in a peer-to-peer manner without assistance of server. However, existing learning frameworks usually assume perfect communication among devices, where they can reliably exchange messages, e.g. , gradients or parameters. But real-world networks are prone packet loss and transmission errors. Transmission reliability comes with price. The commonly-used solution is adopt reliable transportation layer protocol, control protocol (TCP), which however leads significant overhead reduces connectivity that be supported. For network lightweight unreliable user datagram (UDP), we propose robust stochastic gradient descent (SGD) approach, called Soft-DSGD, address unreliability issue. Soft-DSGD updates parameters partially received messages optimizes mixing weights according link matrix links. We prove proposed system, even communications, still achieve same asymptotic convergence rate as vanilla SGD communications. Moreover, numerical results confirm approach leverage all available links speed up convergence.